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The Workforce Investment Act uses measures of labor market outcomes to drive program performance. 

This paper examines the relationship of participant characteristics with key program outcome 

measures and evaluates the impact of different training interventions of program outcomes while 

controlling for participant characteristics. Paper suggests that future performance measures should 

be adjusted for participant characteristics and recommends directions for future research. 
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Policymakers have long wanted publicly funded job training programs to produce tangible results for participants 
in the labor market. During recent decades the Department of Labor has funded a host of studies to attempt to 
measure the impact of training programs (King, 2004). Policy makers brought this focus on performance to the 
creation of the Workforce Investment Act (WIA) in 1998. WIA, which replaced the Job Training Partnership Act 
(JTPA), established 17 performance measures to drive program performance. Most of these measures were labor 
market outcome measures, such as changes in participants’ earnings and the rate at which participants are placed 
in jobs. WIA’s performance policy included several other new features. First, data for labor market measures 
would be collected from unemployment insurance records, rather than traditional phone or mail follow-up 
surveys. Second, the performance goals related to these measures would be set through negotiations between the 
federal government and state agencies. The state agencies would in turn negotiate performance levels with local 
areas. Under JTPA, state and local performance goals were set using a regression model developed by the Federal 
Department of Labor, which considered participant demographics and local labor market conditions. The 
regression model adjusted goals based on the demographics of participants served and the local economic 
conditions, so that local areas or states serving a population that was relatively disadvantaged, or serving an area 
with slower economic growth, would have lower goals. Conversely, local areas in a more advantageous position 
were assigned higher goals. Under WIA, that model was abandoned*. 

The design of the WIA program responded to research findings, mostly from welfare reform studies, that 
seemed to show that “work first’’ was more effective than the more traditional “training and job placement” model 
(King, 2004). The new program created “One-Stop Career Centers” where — in theory — all key programs to assist 
the unemployed, such as WIA, Job Service, local educational agencies, would located together. These One-Stops 
are governed by a local area Workforce Investment Boards (WIBs), with each WIB having one or more One- 
Stops in its system. These One-Stop centers offer “universal access” services so that anyone could come in and 
access resources such as job listings, computers and printers for producing resumes, contacts for local employers, 
etc. People who formally registered in the system can further receive three levels of interventions: 

• core services — essentially job search assistance — which mainly consists of access to information to help 
participants quickly find jobs with minimal staff help; 

• intensive services, in which professional staff provide counseling, skills assessment and some support 
services such child care or transportation subsidies for clients, 

• training services — mostly provided through a voucher system called Individual Training Accounts 
(ITAs) — where participants get traditional skill training and general education. OJT training may also be 
provided. (O’Leary, Straits & Wander, 2004) 

The idea was that all participants first try to find a job their own, and if that was not successful they would 
then be enrolled in WIA and provided staff assistance and more intensive resources. Then, only as a last resort 
would formal training or OJT be provided. 

The federal government has yet to complete a comprehensive evaluation of WIA. An assessment of the early 
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* For a complete history of performance policies in federal employment and training programs see Bamow and 
Smith (2004). 




implementation of WIA performed by Social Policy Research Associates found that in general, the 17 
performance measures made sense to program operators and that the system was focused on performance. 

Problems that were identified with the measures were that there may be too many measures, and the 
definitions of some measures were vague (Social Policy Research Associates, 2004). 

Problem Statement 

WIA has produced a host of publicly available performance data based on its 17 performance measures. At the 
same time, the Federal Department of Labor is moving to a new performance measurement system dubbed 
“common measures” which consists of fewer labor market outcome measures that will cut across a host of federal 
employment programs in addition to WIA (Department of Labor, 2006). Policymakers at the state and local level 
and program managers must manage a system dominated by these measures. Yet, there is very little research on 
the factors that influence a programs performance on these measures. The unspoken assumption of Federal 
policymakers is that good programs will do well on the measures. Yet we know from previous research that 
participant characteristics and particular program interventions affect outcomes significantly. Lacking any well- 
funded, large-scale federal evaluation studies, policymakers and managers are adrift as they struggle to improve 
performance on these measures. 

Theoretical Perspective 

This paper is based on a stream of research stretching back to the 1960s, which uses experimental and quasi- 
experimental designs to evaluate the effectiveness of public training programs by measuring the labor market 
outcomes for participants (see for example Gay & Borus, 1980, Cragg, 19997, Bamow, 2000, Heckman, 
Heinrich& Smith 2002).^ Recent studies in this stream of research include Bloom, et. al.(1997) that used 
experimental designs to evaluate the effectiveness of various program interventions within the JTPA program. 
Leigh (2000) examined the impact of federal programs on Dislocated Workers and Schochet, et.al. 2001 evaluated 
the impact of the federal Job Core program. Moore et. al. (2004) used a quasi-experimental design to assess the 
impact of different types of interventions in a state-funded worker training program in California. These studies 
and many others attempt to measure the impact of programs on participant labor market success while controlling 
for the characteristics of participants. Many studies have found that some groups, such as people with low levels 
or education, disabilities or limited ability to speak English, have a more difficult time succeeding in the labor 
market. Hence, studies often attempt to measure and account for these characteristics in assessing the impact of 
the program. 

Research Questions 

This paper attempts to answer two research questions which are fundamental to understanding the relationship 
between the WIA program and its performance measurement system: 

1 . What participant characteristics are associated with success on key WIA current and future performance 
measures? 

2. After controlling for participant characteristics do the different training interventions have differing 
impacts on earnings or likelihood of employment after service? 

Method and Limits 

Study Population 

In the WIA program. Adult participants are divided into two categories for program services: (a) “Adult” 
participants, who are essentially unemployed, low-income people over 21, and (b) “Dislocated Workers,” who 
have recently been laid off from jobs. Throughout the program, these participants are treated separately and their 
outcomes are accounted for separately. The WIA legislation specifies that performance should be evaluated for 
each “program year,” with a program year essentially being the one-year period that begins each July 1 . In this 
study we analyze the experience of these two groups separately. To create the study population, we took 
electronic records of all Adult and Dislocated Worker participants from a larger urban area in California who 
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exited in program years 2002-03, 2003-04, 2004-05. Complete data were available for 7,913 Adult participants 
and 3,850 Dislocated Worker participants who had exited the WIA program in the study years. 

Data Sources and Measures 

Data came from two sources. First, we obtained administrative records indicating participant characteristics 
and services received. Second, we obtained data on labor market experience — specifically, earnings and 
employment status — resulting from matching participants to the unemployment insurance (UI) base-wage file. 
This match was done by the California Employment Development Department (EDD) to generate the current 
federal performance measures. Since the EDD does not yet calculate the “common measure” referred to in WIA 
as Average Earnings, we further used the data to calculate Average Earnings”.^ Table 1 shows the measures used 
in the study, and descriptive data for the study population. Due to space constraints, we only provide descriptive 
data for the variables that were statistically significant at the p<0.05 level in the final model, though in our models 
we tested many other variables. The measures fall into four categories - (1) the labor market outcomes we will 
try to predict, (2) participant characteristics, and (3) training interventions, and (4) variables that control for the 
participants’ economic context. 



Table 1 . Characteristics and Experiences of the Study Population 



Measure 


Adult 


Dislocated 

Worker 


Labor Market Outcomes Measures 

Average Earnings 


$10,349 


$14,766 


(Earnings 2"“* plus 3'^^ quarter after exit) 
Entered Employment"^ 


75% 


76% 


(Found employed in base wage file Quarter after exit) 

Participant Characteristics 

Limited-English Speaker 


12% 


11% 


Disabled 


14% 


<1% 


Low Income 


68% 


45% 


Ethnicity: Latino 


42% 


41% 


Ethnicity: African American 


31% 


20% 


Ethnicity: Asian 


10% 


10% 


Ethnicity: White 


16% 


28% 


Female 


54% 


51% 


Receiving Public Assistance 


69% 


46% 


Education: Less than Middle School 


5% 


7% 


Education: Did not complete high school 


15% 


10% 


Education: High School Diploma 


51% 


43% 


Education: Some College 


15% 


23% 


Education: Bachelor's Degree 


13% 


19% 


Age at Exit 


35.4 years 


41.9 years 


Training Interventions 

Intervention - Established Individual Training Account (ITA) 


12% 


17% 


Intervention - Received On-the Job Training (OJT): 


10% 


3% 


Intervention - Received Occupational Skills Training: 


12% 


18% 


Context Variables 

Earnings 3rd Quarter Prior to Registration in the program 


$2,615 


$6,593 



^ Given that the federal government is shifting to the new “common measures” we chose to model two measures 
from the new common measures rather than the older WIA measures. The Average Earnings measure is a new 
measure, and is calculated by taking the total earnings of program exiters who are found employed in the 2"^ and 
3"^^ quarter after training, and dividing these total earnings by the number of program participants who exited the 
program in the appropriate quarter. A complete description of the new common measures can be found on the 
United States Department of Labor www site (www.dol.gov). 

Entered Employment is defined in the WIA legislation as a key outcome measure, and is measured as the 
number of program exiters who were found in the base wage file to be employed in the first quarter after exit, 
divided by the number of program exiters. 




Measure 


Adult 


Dislocated 

Worker 


Quarter Exited: The quarter exited was entered into the model as a 
dummy variable to control for seasonality and general economic 
conditions 





As the table shows, this local WIA program serves a relatively disadvantaged population. A large proportion 
of participants were receiving public assistance, and most were classified as “low income” when they entered the 
program. Latino was the dominant ethnic group, followed by African Americans and Whites, and women slightly 
out number men. Dislocated Worker participants tend to be older and more educated than Adult participants. 
Analysis 

We generated four regression models to answer our research questions. First, we used binary logistic 
regression to examine the impact of the variables on Entered Employment rates. Next, we used a linear multiple 
regression model to examine the relationship between the variables and the Average Earning after training. For 
both sets of analyses, we modeled the Adult and Dislocated Worker populations separately. Here we report the 
models that were the best fit in each case. 

Results and Findings 

The model for Entered Employment for Adult participants was highly significant according to the Omnibus test of 
coefficients, though the indicates the model accounted for a modest amount of the variance in the Entered 
Employment rate. The model found significant relationships between a number of participant characteristics and 
the probability of their finding employment when exiting the program. As the results in Table 2 indicate, Asians 
and Latinos were more likely to be employed than other ethic groups. The odds ratio indicates the amount by 
which the likelihood of entering employment changes with a one-point change in the dependent variable. For 
example, the odds of a Latino participant entering employment are 23.4% greater than those of ethnic groups not 
in the equation (for example. Whites). The odds of a “low income” participant entering employment are only 
63.9% as high as the odds of a non-low- income participant entering employment. Put differently, the odds ratio 
for a non-low-income participant entering employment are 1/0.639, or 1.564, which means that the odds of a non- 
low-income participant entering employment are 56.4% greater than a low-income participant. 

Adults with disabilities, those with low incomes, older participants, and those not having a high school 
diploma had a lower likelihood of finding a job after exiting from the program. With each year of participants’ 
age, the odds of entering employment (under the WIA definition) decrease by 1.7% (calculated as 1-0.983). So 
the odds of a participant of 47 years of age (for example) entering employment will be 84.2% (calculated as 0.983 
raised to the tenth power) of that for a 37-year-old participant, on average. Interestingly, none of the training 
interventions were significantly associated, positively or negatively, with finding employment. A significant 
seasonal effect was found for some quarters, with participants exiting in some quarters being significantly more 
likely to find employment and those exited in other quarters less likely to find employment. 

Table 2. Entered Employment Rate, Adult Participants, Binary Logistic Regression Results 



Variable 


Beta 


S.E. 


Wald 


Sig. 


Odds ratio 


Disability 


-0.50 


0.21 


5.83 


0.02 




Low income 


-0.45 


0.09 


26.78 


0.00 




Ethnicity - Latino 


0.21 


0.08 


7.31 


<0.01 


1.23 


Ethnicity - Asian 


0.46 


0.14 


11.28 


0.01 


1.58 


Education: Completed Middle 
School, but not High School 


-0.20 


0.10 


4.12 


0.04 


0.82 


Age at exit 


-0.02 


<0.01 


31.39 


0.00 


0.98 


2002 Q3_exited 


-1.11 


0.46 


5.68 


0.02 


0.33 


2003 Q1 exited 


0.51 


0.16 


10.58 


<0.01 


1.67 


2003 Q4 exited 


0.26 


0.11 


5.28 


0.02 




2004 Q1 exited 


0.53 


0.11 


21.75 


0.00 




Constant 


2.10 


0.16 


184.82 


0.00 


8.20 







N = 10,241, Cox & Snell = 0.026, Nagelkerke = 0.040 
Omnibus test of coefficients (chi=squared) = 128.72 
Omnibus test of coefficients (significance) = 0.00 

The model for predicting the Entered Employment rate for Dislocated Worker participants was also highly 
significant overall but also predicted only a modest amount of the variance in this key outcome. Results are 
shown in Table 3. The variables which proved significant were different than for Adults. Again African 
Americans and Latinos were more likely to be employed than other groups, and previous earning were positively 
associated with the likelihood of finding employment. Age had a negative relationship with finding employment. 
For Dislocated Workers, having been on public assistance had a significant negative impact on the likelihood of 
finding employment. Two training variables were significantly associated with the likelihood of finding 
employment. Having received OJT training was positively associated with finding employment while, having 
received occupational skills training was negatively associated with finding employment. It may be that having 
established a track record with an employer through OJT may make continued employment more likely, while 
those who received occupational skills may have still had some significant skill deficiencies — compared to those 
who received no occupational skills training — following the training. 

Table 3. Entered Employment, Dislocated Worker Participants, Binary Logistic Regression Results 



Variable 


Beta 


S.E. 


Wald 


Sig. 


Odds ratio 


Ethnicity - Latino 


0.41 


0.08 


24.93 


0.00 


1.51 


Ethnicity - African American 


0.31 


0.09 


10.78 


0.00 


1.37 


Public Assistance 


-0.21 


0.07 


8.40 


0.00 


0.81 


Age at exit 


-0.02 


0.00 


33.69 


0.00 


0.98 


OJT Training 


0.78 


0.29 


7.27 


<0.01 


2.19 


Occupational Skills Training 


-0.37 


0.08 


20.73 


0.00 


0.69 


Earnings 2'"^ Qtr Prior to 
Dislocation 


<.01 


<0.01 


4.78 


0.03 


1.00 


2002 Q1 Exited 


0.35 


0.16 


4.60 


0.03 


1.42 


2002 Q2 Exited 


-0.41 


0.09 


19.72 


0.00 


0.67 


Constant 


1.86 


0.17 


119.59 


0.00 


6.44 



N = 4,879, Cox & Snell R" = 0.03, Nagelkerke R" = 0.05 
Omnibus test of coefficients (chi=squared) = 153.01 
Omnibus test of coefficients (significance) = <0.00 

We turn now to the models which looked at the impact of the variables on earning after training. Results are 
shown in Table 4. For Adult participants, our model is highly significant overall and accounts for a considerable 
amout of the variance in earnings - over 23%. Participants having higher levels of education had greater earnings 
after exiting the program.. Not surprisingly, participants’ previous earnings are positively associated with 
earnings after exit from the program, indicating that those that earned more in the past will earn more in the 
future. A number of characteristics had a negative association with Average Earnings, including having limited 
proficiency in English, being disabled, being Latino or African American, female or being on public assistance. 
Again we found some limited effects of seasonality or economic conditions,. Also, as with the Adult Entered 
Employment model, we found no significant relationship, positive or negative, between any training intervention 
and earnings. 

Taking this model in conjunction with the Entered Employment model (Table 2), some interesting 
interactions arise. While Latinos are more likely than most others to find a job after exiting the program. Table 4 
suggests that those jobs are at lower wages. And while Table 4 not surprisingly shows that participants having a 
higher degree of educational attainment also earn more. Table 2 further shows that they are more likely to find a 
job. In other words, participants of low educational attainment take a double hit - lower likelihood of finding a 
job, and less pay when they do. 




Table 4. Average Earnings, Adult Participants, Regression Results 



Variable 


B 


S.E. 


T 


Sig. 


(Constant) 


9,489.33 


260.58 


36.41 


0.00 


Limited English Speaker 


-543.34 


212.51 


-2.55 


0.01 


Disability 


-1,630.74 


550.94 


-2.96 


<0.01 


Ethnicity: Latino 


-741.55 


171.30 


-4.32 


0.00 


Ethnicity: African American 


-934.92 


175.69 


-5.32 


0.00 


Female 


-608.97 


129.32 


-4.70 


0.00 


Public Assistance 


-1,202.71 


147.60 


-8.14 


0.00 


Education: High School Diploma 


588.84 


179.89 


3.27 


0.00 


Education: Some College 


1,451.19 


234.06 


6.20 


0.00 


Education: Bachelor's Degree 


3,862.74 


253.27 


15.25 


0.00 


Earnings 3rd Quarter Prior to Registration 


0.60 


0.02 


33.55 


0.00 


2002 Q4 exited 


-1,253.94 


270.91 


-4.62 


0.00 


2003 Q3_exited 


856.20 


288.84 


2.96 


0.00 


2003 Q4_exited 


899.82 


276.11 


3.25 


0.01 


2004 Q1 exited 


938.40 


254.49 


3.68 


0.00 



Adjusted R" = 0.230, F = 170.305, N = 7,913 



In the case of Dislocated Workers our model shows very different results. Again the model if highly 
significant and robust in that it accounts for over 30% of the variance in post-program earnings. The effects of 
demographic variables are similar to those in the Adult Participant model. Latino, African American, and female 
participants tend to earn less as do older participants. Those with higher levels of education earn more. 
Participants classified as “low income” on entering the program tend to have lower earnings after exiting the 
program. Prior earnings are positively associated with post-training earnings. In the case of Dislocated Workers, 
training was significantly related to earnings. Participants having established an ITA — a voucher which pays for 
training at some approved training institution — earn an average of over $12,000 less in the two quarter follow-up 
period compared to other trainees who got other training services or no training. Conversely, having received 
occupational skills training, which indicates longer training in a specific occupation, is associated with a large 
increase in earnings in this model - more than $ 1 1 ,000 extra over the 6-month follow-up period. 

Taking this model in conjunction with the Entered Employment model (Table 3), some interactions arise. 
While Latino and African American Dislocated Workers are more likely than most others to find a job after 
exiting the program, those jobs are at lower wages. And older Dislocated Workers are less likely to find a job, 
and when they do find a job they earn less money than younger workers. While participants receiving 
“occupational skills” training earn more money than those who did not, they also, for reasons that are not clear, 
have a lower likelihood of finding a job. 

And while Table 4 not surprisingly shows that participants having a higher degree of educational attainment 
also earn more. Table 2 futher shows that they are more likely to find a job. In other words, participants of low 
educational attainment take a double hit - lower likelihood of finding a job, and less pay when they do. 








Table 6. Average Earnings, Dislocated Worker, Regression Results 



Variable 


B 


S.E. 


T 


Sig. 


(Constant) 


12,644.60 


789.69 


16.01 


0.00 


Low Income 


-1,277.54 


276.77 


-4.61 


0.00 


Ethnicity: Latino 


-1,511.58 


338.57 


-4.46 


0.00 


Ethnicity: African American 


-2,119.31 


372.56 


-5.68 


0.00 


Female 


-1,690.01 


261.93 


-6.45 


0.00 


High School Diploma 


1,386.90 


399.21 


3.47 


0.00 


Some College 


2,818.730 


463.06 


6.08 


0.00 


Bachelor's Degree 


5,226.42 


500.19 


10.44 


0.00 


Age at Exit 


-40.06 


12.52 


-3.20 


0.00 


Intervention - Established ITA 


-12,398.91 


4,644.18 


-2.67 


<0.01 


Intervention - Received Occupational Skills Training 


11,247.52 


4,638.82 


2.42 


0.01 


Earnings 3rd Quarter Prior to Registration 


0.67 


0.02 


29.06 


0.00 


y2002_ql_exited 


1,102.04 


548.18 


2.01 


0.04 


y2002_q4_exited 


-1,481.25 


580.22 


-2.55 


0.01 


y2003_q2_exited 


-1,158.92 


423.92 


-2.73 


<0.01 



Adjusted R" = 0.323 F = 132.348 N = 3,850 



Conclusions and Recommendations 

These are exploratory results and it important that we do not over-interpret them, but a few important findings 
stand out and deserve consideration by policy makers and further research by analysts. First, as in most research 
on the labor market outcomes of training, demographics matter. Participants who are disadvantaged in the labor 
market because they are disabled, have limited English proficiency, are older or on public assistance are unlikely 
to have good success in the labor market regardless of what services they receive. This suggests to us that a 
system of adjusting performance standards for states, local area and agencies that serve more disadvantaged 
populations is rational and would make for good public policy. Other analysts who have examined the 
implementation of WIA agree (Siedlecki & King, 2005). Adjusting performance for the demographics removes 
the disincentive to serve the most disadvantaged who need the services most. The seems an essential policy to us 
if the program is to reach its target population of disadvantaged workers and effectively move people from 
dependence on public assistance to self support. Various forms of regression modeling with national populations 
provide a good tool for guiding these adjustments. 

Second, our results suggest that training is not a universally effective intervention for participants. This 
finding could cause one to question the extent to which One-Stops have been able to effectively assess 
individuals’ training needs and prescribe specific training interventions that will lead to good labor-market 
outcomes for program participants. Our data suggest relatively longer term and more expensive training services 
are not uniformly superior to shorter-term, less costly interventions such as job search assistance and one-on-one 
coaching and counseling. Our data suggests that On-The-Job training where employers are offered incentives to 
employ program participant does lead to a significant increase in post program employment, but that some other 
types of training have no affect or a negative affect. Additionally, we found that OJT improves participants’ 
earnings after training.. We found no positive impact for any form of training on Adult WIA participants, which 
is by far the largest group in the program. For Dislocated Workers occupational skill training appeared to have a 
strong positive effect on earnings but, other forms of training was associated with lower earnings. 

On the other hand, it seems to us that our findings probably hide much more complex relationships. The 
effects of training are probably governed by relationships our models could not measure. For example, it would 
be interesting and likely meaningful to discover the reason(s) why Dislocated Workers receiving “occupational 
skills’’ training have higher earnings, but a lower likelihood of finding a job. Experience suggests that factors 
such as the quality of training, the appropriateness of training for the local labor market, the motivation and ability 
of the participant are all key to understanding the impact of training. Still, it seems policymakers should be 
somewhat skeptical about the value of expensive training interventions in WIA. There is a need for researchers to 
undertake much more nuanced studies of the connections between training and labor market outcomes in WIA, 
with carefully controlled studies to identify the types of participants and types of training most likely to lead to 






successful outcomes. This will undoubtedly require a mix of qualitative studies, experimental studies and large 
quasi-experimental analyses such as this one to hone in the value of training in WIA. 

Finally, since our model has found that WIA outcome measures are related to demographic factors in some 
important ways, it will be interesting to see whether or not the demographics of participants change as the WIA- 
mandated outcome measures change. One-Stop operators and local area WIBs both have some incentive to 
perform well on the outcome measures, and when outcome measures can be affected simply by adjusting the 
demographic profile of those served, it would not be surprising if One-Stops and local area WIBs were to shift 
their resources toward serving those more likely to help improve their outcome measures. While such a shift in 
demographics of those served could be reasonably expected to occur, it seems that at least some policy-makers 
would consider a shift toward a more “advantaged” demographic profile to be neither desirable nor acceptable. 
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